1 link tagged with all of: evaluation + japanese + benchmark + language understanding + framebench
Click any tag below to further narrow down your results
Links
This arXiv paper introduces FrameBench, a multiple-choice benchmark that checks whether language models can tell apart the semantic frames a verb evokes in different sentences, the implicit background inferences humans make when reading. It covers English and Japanese, built from FrameNet-style resources with native-speaker verification.
- The benchmark tests whether models distinguish the frames the same verb evokes across contexts, probing implicit enrichment rather than surface-level language skill.
- Small models struggle noticeably, while several large models score above the human reference baseline.
- The dataset and construction and evaluation code are publicly released, so the results can be reproduced and extended.